Skip to content

Add docs for UI Agents#772

Open
markbackman wants to merge 4 commits into
mainfrom
mb/ui-agent
Open

Add docs for UI Agents#772
markbackman wants to merge 4 commits into
mainfrom
mb/ui-agent

Conversation

@markbackman
Copy link
Copy Markdown
Contributor

@markbackman markbackman commented Apr 25, 2026

Documentation update for the UI Agent Protocol that shipped across these four PRs:

  • pipecat-ai/pipecat#4407 — wire format as first-class RTVI message types (5 new message types, paired Data/Message envelopes, 5 new pipeline frames, on_ui_message handler, payload models, PROTOCOL_VERSION 1.2.0 → 1.3.0).
  • pipecat-ai/pipecat-client-web#203UIAgentClient plus React idioms (UIAgentProvider, UITasksProvider, hooks, standard handlers, A11ySnapshotStreamer).
  • pipecat-ai/pipecat-ai-subagents#18UIAgent (now extends LLMContextAgent), expanded action helpers, ReplyToolMixin, start_user_task_group for streaming worker fan-out, keep_history flag, attach_ui_bridge rewired to on_ui_message.
  • pipecat-ai/pipecat-music-player#1 — voice-driven music player as the canonical reference app.

The wire format moved into Pipecat as the canonical authority, so single-LLM apps can target it directly without taking a subagents dependency. Subagents builds the agent abstractions on top.

Changes

Pipecat (server)

  • client/rtvi-standard.mdx — bumped version note to 1.3, added a UI Agent Protocol section documenting all 5 message types (ui-event, ui-snapshot, ui-cancel-task, ui-command, ui-task) with shapes, plus a Standard Command Payloads table for the 8 standard payload models.
  • api-reference/server/rtvi/rtvi-processor.mdx — added a UI Agent Protocol section documenting the new on_ui_message event handler and the inbound / outbound frame conventions.
  • api-reference/server/rtvi/rtvi-observer.mdx — extended the frame-translation table for RTVIUICommandFrame and RTVIUITaskFrame.
  • api-reference/server/frames/system-frames.mdx — added a RTVI UI Frames section documenting all 5 new pipeline frames (RTVIUICommandFrame, RTVIUITaskFrame, RTVIUIEventFrame, RTVIUISnapshotFrame, RTVIUICancelTaskFrame).

Subagents

  • api-reference/pipecat-subagents/ui-agent.mdx — fixed parent class to LLMContextAgent, refreshed bridge frame references, added keep_history, added the 5 action helpers (scroll_to, highlight, select_text, click, set_input_value), added user_task_group / start_user_task_group / reset_context, added on_task_cancelled lifecycle hook, documented single-flight task lock semantics.
  • api-reference/pipecat-subagents/ui-tools.mdx — full rewrite for ReplyToolMixin (the prior ScrollToToolMixin / HighlightToolMixin documentation no longer matched the source).
  • api-reference/pipecat-subagents/ui-commands.mdx — fixed import path to point at pipecat.processors.frameworks.rtvi.models (canonical home), added Click, SetInputValue, SelectText.
  • api-reference/pipecat-subagents/ui-prompts.mdx — fixed import path, added <selection> block bullet.
  • api-reference/pipecat-subagents/messages.mdx — added 4 task-lifecycle bus messages (BusUITaskGroupStartedMessage, BusUITaskUpdateMessage, BusUITaskCompletedMessage, BusUITaskGroupCompletedMessage), refreshed attach_ui_bridge to reflect the new wiring (subscribes to on_ui_message, emits RTVIUICommandFrame / RTVIUITaskFrame), removed stale UI_*_MESSAGE_TYPE constants section.
  • api-reference/pipecat-subagents/overview.mdx — added LLMContextAgent row to the agent hierarchy and fixed UIAgent's parent.
  • NEW: subagents/learn/ui-agent.mdx — consolidated UI Agent guide. Sections: Why, Architecture (voice + UI agent split), How information flows (4 loops), Sequence diagram (voice + UI for action), Action vocabulary (pointing / reading / writing / presentation), Choosing a tool shape (ReplyToolMixin vs custom @tool reply vs per-domain tools), Async tasks and lifecycle (start_user_task_group + useUITasks + cancel), When to use (deployment shapes), Reference app, API reference. Sourced from the internal design doc and the music-player demo.
  • subagents/examples/overview.mdx — added a Reference Applications section linking pipecat-music-player.
  • docs.json — registered subagents/learn/ui-agent in the navigation.

Client (js + react)

  • api-reference/client/js/ui-agent-client.mdx — refreshed to RTVI v1.3 (RTVIEvent.UICommand instead of RTVIEvent.ServerMessage, kebab-case wire types), added addTaskListener / removeTaskListener / removeAllTaskListeners / cancelTask.
  • api-reference/client/js/a11y-snapshot.mdx — replaced the dropped UI_SNAPSHOT_EVENT_NAME section with A11ySelection shape docs and a See-also section noting the streamer dispatches via RTVIMessageType.UI_SNAPSHOT.
  • api-reference/client/js/client-methods.mdx — added sendRTVIMessage(msgType, data?) section under Messages.
  • api-reference/client/react/hooks.mdx — added useStandardSelectTextHandler, useStandardSetInputValueHandler, useStandardClickHandler, useUITasks; updated useStandardCommandHandlers to reflect that it installs all six handlers.
  • api-reference/client/react/components.mdx — added UITasksProvider; refreshed UIAgentProvider description for RTVIEvent.UICommand / RTVIEvent.UITask.

Verification

  • mint broken-links — no broken links found across the entire docs site.
  • npx prettier --write — applied to all touched files.

@mintlify
Copy link
Copy Markdown

mintlify Bot commented Apr 25, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
daily 🟢 Ready View Preview Apr 25, 2026, 1:08 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

Updates the docs to align with the UI Agent Protocol shipped across
pipecat#4407, pipecat-client-web#203, and pipecat-ai-subagents#18.
The wire format moved into pipecat as canonical (5 first-class RTVI
message types, paired Data/Message envelopes, 5 new pipeline frames,
PROTOCOL_VERSION 1.2.0 → 1.3.0); subagents builds the agent
abstractions on top; client-web ships UIAgentClient + React idioms.

Pipecat (server)

- client/rtvi-standard.mdx: bumped version note to 1.3, added a UI
  Agent Protocol section documenting all 5 message types (ui-event,
  ui-snapshot, ui-cancel-task, ui-command, ui-task) with shapes and
  the 8 standard payload models.
- api-reference/server/rtvi/rtvi-processor.mdx: added a UI Agent
  Protocol section documenting the on_ui_message handler and the
  inbound / outbound frame conventions.
- api-reference/server/rtvi/rtvi-observer.mdx: extended the
  frame-translation table for RTVIUICommandFrame and RTVIUITaskFrame.
- api-reference/server/frames/system-frames.mdx: added a RTVI UI
  Frames section documenting all 5 new pipeline frames.

Subagents

- api-reference/pipecat-subagents/ui-agent.mdx: fixed parent class
  (LLMContextAgent), refreshed bridge frame references to
  RTVIUICommandFrame / RTVIUITaskFrame, added keep_history param,
  added 5 action helpers (scroll_to / highlight / select_text /
  click / set_input_value), added user_task_group /
  start_user_task_group / reset_context methods, added
  on_task_cancelled lifecycle hook, documented single-flight task
  lock semantics.
- api-reference/pipecat-subagents/ui-tools.mdx: full rewrite for
  ReplyToolMixin (the prior ScrollToToolMixin / HighlightToolMixin
  documentation no longer matched the source).
- api-reference/pipecat-subagents/ui-commands.mdx: fixed import path
  to point at pipecat (canonical home), added Click, SetInputValue,
  SelectText.
- api-reference/pipecat-subagents/ui-prompts.mdx: fixed import path,
  added <selection> block bullet.
- api-reference/pipecat-subagents/messages.mdx: added 4 task-lifecycle
  bus messages (BusUITaskGroupStartedMessage, BusUITaskUpdateMessage,
  BusUITaskCompletedMessage, BusUITaskGroupCompletedMessage),
  refreshed attach_ui_bridge to reflect the new wiring (subscribes to
  on_ui_message, emits RTVIUICommandFrame / RTVIUITaskFrame), removed
  stale UI_*_MESSAGE_TYPE constants section.
- api-reference/pipecat-subagents/overview.mdx: added LLMContextAgent
  row to the agent hierarchy and fixed UIAgent's parent.
- subagents/learn/ui-agent.mdx: NEW consolidated guide. Sections:
  Why, Architecture, How information flows, Sequence diagram, Action
  vocabulary, Choosing a tool shape, Async tasks and lifecycle, When
  to use, Reference app, API reference. Sourced from the internal
  design doc and the music-player demo.
- subagents/examples/overview.mdx: added a Reference Applications
  section linking pipecat-music-player.
- docs.json: registered subagents/learn/ui-agent in the navigation.

Client (js + react)

- api-reference/client/js/ui-agent-client.mdx: refreshed to RTVI v1.3
  (RTVIEvent.UICommand instead of RTVIEvent.ServerMessage, kebab-case
  wire types), added addTaskListener / removeTaskListener /
  removeAllTaskListeners / cancelTask, added See-also pointers.
- api-reference/client/js/a11y-snapshot.mdx: replaced the dropped
  UI_SNAPSHOT_EVENT_NAME section with A11ySelection shape docs and a
  See-also section noting the streamer dispatches via
  RTVIMessageType.UI_SNAPSHOT.
- api-reference/client/js/client-methods.mdx: added sendRTVIMessage
  section under Messages.
- api-reference/client/react/hooks.mdx: added useStandardSelectTextHandler,
  useStandardSetInputValueHandler, useStandardClickHandler, useUITasks;
  updated useStandardCommandHandlers to reflect that it installs all
  six handlers.
- api-reference/client/react/components.mdx: added UITasksProvider;
  refreshed UIAgentProvider description for RTVIEvent.UICommand /
  RTVIEvent.UITask.

Verified with mint broken-links (no broken links found) and
prettier --write across all touched files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant